LLM Inference

4 posts tagged with "LLM Inference"

August 12, 2026

Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

边缘/MoE 推理实证：OLMoE-1B-7B 在 Jetson Orin Nano 上比同 active 参数量 Dense 模型慢 31%，能耗高 2.1 倍；边缘带宽受限硬件上推理成本跟随总参数而非激活参数

MoE Edge Inference LLM Inference Hardware Benchmark Empirical Study

August 12, 2026

[论文精读] OasisKV: Scaling In-Decode KV Cache

A memory-centric LLM inference system that decouples full KV-cache storage from HBM, using lookahead tokens from speculative decoding to prefetch only the most relevant KV blocks, achieving 1.69×-2.3× throughput gains within 0.7 points of full-attention accuracy.

KV Cache Sparsity LLM Inference Sparse Attention Speculative Decoding Prefetching Disaggregated Serving Memory Management

July 23, 2026

Robust KV Cache Management for LLM Serving under Output Token Length Uncertainty

提出了一种基于Wasserstein DRO的鲁棒KV cache管理框架，联合优化GPU并行配置、KV cache预留、请求路由和前缀缓存，在输出token长度不确定性下实现自适应内存分配和尾延迟控制。核心贡献包括临界分位数结构理论证明、BCD-DRO分解算法和滚动时域自适应策略。

LLM Serving KV Cache Distributionally Robust Optimization Wasserstein DRO GPU Resource Management Queueing Theory LLM Inference

May 1, 2026

TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference

首个在低延迟推理场景下为张量并行LLM推理实现高效计算通信重叠的系统，通过融合AllReduce-RMSNorm核和波感知智能分割达成最高1.28×加速

LLM Inference Tensor Parallelism Compute-Communication Overlap CUDA Kernel Fusion NVSHARP Multimem RMSNorm vLLM